fix(api): price cached input tokens at the cached rate - #5965

Summary

Traced cost bills every prompt token at the full input rate, including the slice a provider served from its cache. On the case in #5711 — a 25,978-token Gemini prompt of which 24,540 were cached — that reports $0.007793 of prompt cost where $0.001168 is correct: 6.67x too high. It runs in OSS as well as cloud, and it is worst on agent workloads, which replay a long prefix on every call.

calculate_costs read only prompt and completion out of tokens.incremental and called litellm's cost_per_token with those alone — even though that function accepts cache_read_input_tokens and litellm's price map carries a separate, much lower cached rate (a tenth of input, for Gemini Flash).

One correction to the issue

The issue says the cached count is never recorded. That is true of the SDK path but not of the OTLP path: logfire_adapter.py already maps gen_ai.usage.cache_read.input_tokens → ag.metrics.unit.tokens.cache_read, and span_data_builders.py rewrites unit.tokens.* → tokens.incremental.*, so it does land in ag.metrics.tokens.incremental.cache_read. The runner already emits that attribute. The count was being recorded and then dropped on the floor at the one place that prices tokens.

Three things this has to get right

1. The cached count is a subset of prompt_tokens, not an addition to it. litellm's convention is that prompt_tokens already includes the cached slice (it normalizes Anthropic-style usage, where the input count excludes them, on the way in). So the count is passed alongside the prompt total, never deducted from it — deducting would understate cost instead of overstating it.

2. It must arrive as an int. This one is not obvious and it is the whole fix. litellm reads the slice back off Usage.prompt_tokens_details.cached_tokens, and its Usage model only derives that wrapper from an int:

Usage(prompt_tokens=25978, cache_read_input_tokens=24540)     # int
  -> prompt_tokens_details=PromptTokensDetailsWrapper(cached_tokens=24540)

Usage(prompt_tokens=25978, cache_read_input_tokens=24540.0)   # float
  -> prompt_tokens_details=None

Agenta stores token metrics as floats, so passing the value through as-is leaves prompt_tokens_details at None and every token is billed at the full input rate again — silently, with no exception to catch. I had this wrong at first; the end-to-end demo below is what caught it, since the mocked unit tests happily showed the kwarg being passed.

3. The adapters disagree on the field name. logfire_adapter writes cache_read (matching what the runner emits), vercelai_adapter writes cached (from ai.usage.cachedInputTokens). Both are read, or the cost is right for one integration and overstated for the other.

Backward compatibility

The kwarg is passed only when the count is non-zero, so a span with no caching calls exactly the signature it always did. The SDK pins litellm>=1,<2, and on a 1.x old enough to lack the parameter an unconditional kwarg would raise TypeError straight into the existing bare except — dropping costs for every span, not just cached ones.

SDK side

The litellm handler never recorded the cached count, so cache_read could not reach the API for SDK-traced calls. It now reads prompt_tokens_details.cached_tokens (OpenAI and Google both report it there) with a fallback to a flat cache_read_input_tokens (Anthropic-style). The extraction was duplicated between the sync and async paths; it is now one helper.

Closes #5711

Testing

Verified locally

Run on Linux (python:3.11-slim container) because litellm 1.92.0 ships manylinux wheels only.

• The demo below is real captured output — real litellm 1.92.0 price map, real calculate_costs, no mocks. The BEFORE line is literally the call the old code made.
• pytest oss/tests/pytest/unit/tracing (api) — 92 passed.
• pytest oss/tests/pytest/unit (SDK) — 2022 passed, 4 skipped, 10 xfailed.
• python run-tests.py --layer unit (api) — 2210 passed, 4 failed. The 4 failures are in test_web_entrypoint_email_env.py (SMTP entrypoint config) and are pre-existing: they fail identically on a clean checkout of main with this branch's changes stashed. Nothing this PR touches is involved.
• ruff format --check and ruff check on all four changed files — clean.
• Confirmed the new tests fail without the fix: reverting trees.py alone fails exactly the three cached-behaviour tests, while the two "signature unchanged" tests keep passing in both directions.

Added or updated tests

api/oss/tests/pytest/unit/tracing/utils/test_trees.py — the _span helper takes an optional cached count under a configurable key, plus:

• test_calculate_costs_passes_cached_tokens_to_the_pricer — parametrized over cache_read and cached, asserting the count is forwarded and the prompt total is passed through untouched.
• test_calculate_costs_sends_the_cached_count_as_an_int — the float trap in point 2. Without the coercion the fix silently does nothing, so this is the test that matters most.
• test_calculate_costs_omits_cache_kwarg_when_nothing_was_cached — calls a pricer whose signature accepts nothing else, so a regression to an unconditional kwarg fails loudly.
• test_calculate_costs_ignores_a_zero_cached_count — an explicit zero is a cache miss.
• test_calculate_costs_bills_cached_tokens_below_fresh_input — end-to-end ratio against a pricer modelling litellm's published contract.

sdks/python/oss/tests/pytest/unit/test_litellm_token_usage.py (new) — 8 cases over _extract_token_usage: OpenAI-style objects, dict-shaped usage, the flat Anthropic-style field, precedence when both are present, absent/zero/null cache details, and a response with no usage object.

QA follow-up

Worth a maintainer's eye on two things I deliberately left alone:

• Cumulation. cumulate_tokens has a hardcoded prompt/completion/total shape, so cache_read does not roll up to parent spans. Costs are correct either way (they cumulate from the corrected per-span costs); this only means the cached token count is not visible in tokens.cumulative. Extending it changes the rollup shape the frontend reads, which felt out of scope here.
• Field-name drift. Reading both cache_read and cached fixes the symptom. Normalizing vercelai_adapter to emit cache_read would fix the cause, but it changes ingest behaviour for existing spans, so I left it.

Also worth noting: cache_creation is recorded by logfire_adapter and still not priced. Cache writes bill above the normal input rate on some providers, so that is a separate, smaller understatement — happy to follow up if you want it in scope.

Demo

Backend-only change, so the demo is captured terminal output rather than a UI recording: the real calculate_costs against the real litellm price map, before and after, plus a check that an uncached span is unchanged.

Checklist

• I have included a video or screen recording for UI changes, or marked Demo as N/A
• Relevant tests pass locally
• Relevant linting and formatting pass locally
• I have signed the CLA, or I will sign it when the bot prompts me